Transacting “live” data (as opposed to bulk loading a batch of data) into a relational database is generally performed by a client application. Traditionally, this has been a person at a computer, typing in a transaction of some sort, perhaps a customer order. In this scenario, a client application (a web application, for example) invokes an operation (a transaction) on a database to insert, update or delete a record. Today, it is often the case that machines generate these transactions. Consider, for example, processing website click streams, or smartphone/device location-based processing, or smart grid electric meter periodic updates. These real-time data feeds happen automatically and continuously, without human intervention.
Modern systems that ingest (process) these events use queuing systems and have connectors that deliver events to “sinks,” generally some form of data store(s). These queuing systems act as clients to the data store, invoking network (client/server) operations to deliver the event to the data store. A key characteristic of these systems is that both the queuing system and the data store are highly available. As events are being continuously delivered, they must be continuously consumed, even when parts (machines) of the queuing system or parts (machines) of the data store suffer failures (e.g., crash, become unavailable due to network connectivity issues, etc.). However, these intermediate subsystems that act as a clients to a data store result in inefficiencies (e.g., an intermediate hop to the data store is often required, adding time before the data can be processed) and create additional points of failure.